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ABSTRACT 



This paper describes issues encountered in a series of 
large-scale implementations of instructional technology in schools and 
vocational centers across the nation. Three tutors, the Word Problem Solving 
Tutor (WPS) , MAESTRO, the Writing Tutor, and the science tutor. Instruction 
in Scientific Inquiry (ISIS) were developed. These systems were evaluated in 
field studies involving 40-50 teachers and as many as 3,000 students each 
year from 40 public schools and several vocational centers. The studies 
evaluated the effectiveness of the software in enhancing critical thinking 
skills. Findings are discussed in terms of planning for technology (selecting 
technologies, cost of implementation of curriculum, preparing teachers, 
teacher training and integrating technology) ; implementing technology in the 
classroom (educational setting, software, technological personnel support, 
teacher, administrator and vendor/research project responsibilities, and 
technical support) ; and evaluating the effectiveness of technology. (AEF) 
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Abstract: Computer and communication technologies afford unique opportunities for 
teaching and learning. As a result, these technologies have become common, if not 
pervasive, in the modem classroom. Unfortunately, simply providing schools and 
teachers with technology is not enough to ensure appropriate and effective use. This 
paper describes issued encountered in a series of large-scale implementations of 
instructional technology in public schools and vocational centers across the nation. 



Introduction 

Your classroom is equipped with new computers. They are loaded with the latest and greatest 
software titles. Your building is wired for the Internet. This paper discusses the practical issues of 
choosing and using educational software in the classroom. 

Computer and communications technologies provide teachers with a myriad oftools for 
connecting and educating their students. As a result, these technologies have become common in the 
modem classroom. In 1983, schools had one computer for every 125 students. In 1995, schools had one 
computer for every 9 students.(Glennan & Melmed, 1996). Simply putting computers in schools, however, 
does not ensure the effective use of technology. Not surprisingly, several common mistakes are made 
when "wiring" the schools of tomorrow. This paper seeks to inform teachers and administrators about the 
practical issues involved in choosing and using technologies in the classroom from the experience of eight 
years of the Fundamental Skills Training (FST) project. 

Background 

Since 1990, the Air Force Research Lab and the University of Texas at San Antonio, have been 
engaged in a long-term research project to bring state-of-the-art intelligent tutoring technology to bear on 
our nation’s growing literacy skills problem in areas such as mathematics, writing and science. The primary 
goals of the FST project are to design, develop, evaluate and transfer prototype intelligent tutoring systems 
(ITS) to public schools, and, when appropriate, to industry under federal technology transfer guidelines. 
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Three tutors, the Word Problem Solving Tutor (WPS), MAESTRO, the Writing Tutor, and the 
science tutor, Instruction in Scientific Inquiry Skills (ISIS) were developed. These three tutoring systems 
have been evaluated in field studies involving 40 public schools and several vocational centers across the 
nation. These studies, involving 40-50 teachers and as many as 3,000 students each year, have evaluated 
the effectiveness of the software in enhancing critical thinking skills. In addition to the 18 studies 
conducted, a host of information has been gathered concerning integrating technology into traditional 
classrooms. 



Planning for Technology 

Instructional technologies, both hardware and software, cannot be simply purchased and dropped 
into schools for immediate deployment. Rather, administrators and teachers must carefully plan for the 
implementation of technology. 



Selecting Technologies 

Decision-makers must address a few key questions before purchasing educational software. For 
example, how well does the software address student and teacher needs? What does the software do (i.e., is 
it a tutor in a box, a cognitive tool, or a vehicle for student-teacher communication and collaboration)? 

Barbara Means (1994) classifies the use of educational technologies into four broad uses: tools, 
communication, tutors, and exploration. The most simplistic of these categories are educational 
technologies that are used only as tools. A prime example of this would be a word processor. In this sense 
the tool is only a mechanism to assist a student in completing work. Educational technologies that are used 
for communication include programs that allow teachers and students to transmit information to each other 
via a network. Technology used as tutors include programs that teach users information and may include 
demonstrations, real-time analysis or simulations. Technology used to explore would guide the student 
through information allowing the student to learn facts, concepts, procedures, and strategies, as he/she 
interacts with the system (Means, 1 994). 

The extent of the software uses should govern the decision of choosing the software to be used for 
instruction. In addition, other variables may also need to be observed (i.e. What other departments could 
use this software? Do other departments have the software?). 

When evaluating software an important aspect to consider is the role of the software, e.g. Is it to 
be used as a part of the core curriculum or as an enhancement? Does the software serve a dual purpose, 
does it incorporate tools into the tutor or exploration educational technology? Dual functionality is an 
important aspect in the design of any software but evaluators must decide if there are other software 
packages that present the information more efficiently. If so, the evaluators should choose the software that 
presents the most exemplary material and then look to over abundance of word processors that exist to fill 
the void. Remember, the purpose of educational technology is to improve a student's performance, not to 
provide state of the art software. 



Cost of Implementation of Curriculum 

More important and more of a burden than the financial costs are the curricular “costs” associated 
with technology. Teachers must decide what topics they must cut fi*om their normal classroom curriculum 
in order to allocate time to using the software. In some cases, the choice to integrate “learning 
technologies” or educational software to teach specific topics or sub-topics is easy. In other cases it can be 
much more difficult. Unfortunately, many programs are seldom a good match to a particular classroom's 
core curriculum and hence, tend to be regarded only as "enrichment" (Means, 1994). As a result, many 
technologies that assist students in learning have a tendency to have a limited impact on students' core 
educational experiences. Instructors must not only decide how much of a role technology will play in their 
curriculum but also the impact of the programs. However, eliminating 15-20 hours of in-class time over an 
academic year requires the teachers to seriously reconsider their curricular choices. These choices are not 
easy given the pervasive emphasis on state testing requirements but the teacher must decide whether or not 
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the lessons taught by the software are important enough to warrant a significant amount of their class time 
or if the program will only be used as an ’’enrichment.” 



Preparing Teachers 

Research has found that the inability of teachers to effectively use software is, in part, due to the 
lack of resources supporting their use of it. Becker, (1994b) using a survey instrument, found low levels of 
computer use and low levels of effective use. Specifically, teachers were only using computers for drill and 
practice and avoiding more complicated applications. Further, Becker notes that schools which offer high 
levels of teacher development on computers and provide technological coordinators to provide 
technological support are more likely to have teachers using computers effectively. 

Another important question is concerned with how teachers will accept technology. If their 
perception is negative but they have an open mind, time should be spent educating them on the benefits of 
technology. Some teachers may feel apprehensive or may experience fear when using computers. Rosen 
and Weil (1995) found that the principal reason instructors were afraid of technology was due to a 
deficiency of experience using computers. Researchers have documented the critical role of professional 
development in improving teachers’ ability to carry out new approaches to instruction (Collins, 1992; 
Duffy, 1992; El-Dinary & Marks, 1992). Thus, the need for training is evident, teachers must be trained 
and supported before they will feel ready to effectively use the software. 

Generally, a good rule of thumb to follow when it comes to operation of powerful and dangerous 
equipment is that time should be spent training the potential user. Technology is no different. Time and 
effort must be carefully spent to ensure that the user can not only confidently use the software but that 
he/she has developed an in-depth understanding of it and can use it as a tool to teach students. 



Teacher Training 

When implementing technology, the importance of teacher training as a key component cannot be 
emphasized enough. Simply, it is required for successful implementation of any type of technology. Many 
teachers do not have experience using computers to deliver instruction to students. Instead, they use 
computers for recording grades and to develop paper-based handouts and exams. Teachers need not only 
understand instructional software from a user’s point of view, but also how to teach with it. The teacher’s 
role in the classroom is not to put hands on the keyboard or mouse, but to stand next to the students acting 
as a partner in the instructional process. Due to this importance, teachers should receive the time needed to 
attend training and to become effective with the software. 



Integrating Technology 

Perhaps many teachers view using technology as a simplistic tool to assist students with their 
lessons (e.g. a typewriter) instead of utilizing software to explore, to communicate, or to teach students by 
providing information, demonstrations, or simulations. As a result assignments may result in computer use, 
but these activities are less than effective. As a preventative measure, instructors should ask several 
questions before looking to insert technology into their classrooms (i.e. Is this subject appropriate for 
technology? What kind of software exists to support the students' work in extended, authentic learning 
activities? Does this software provide examples and tools to further the students' work?). 

While technology exists in many different forms, it should never be used when it is not 
appropriate. There are many tasks that exist that could be taught using technology, but it should not be 
viewed as always being applicable for teaching. Teachers should choose to use technology based on case 
by case objectives. For example, the use of technology in a History class can be effectively used as 
software can furnish hyperlinks to provide students with information. Other domains, such as Math and 
Science, are ripe for teaching domain knowledge using Computer Based Instruction (CBI) and simulations. 
Conversely, English is a subject that is more difficult to teach using technology. Natural language 
processing is an area that has proved difficult to master. 
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When a teacher or administrator views the domain as appropriate for technology he/she must next 
investigate what types of software are available to assist them in teaching the subject and then evaluate the 
software. Decisions must also be made concerning how the technology will be inserted into the curriculum 
(i.e. Will computer use result in a change of teaching style? Is this software appropriate for this population? 
How will computer use be graded? How much time will be spent on the computer? How will the students 
be introduced to the computers? What new materials will need to be created?). 



Implementing Technology in the Classroom 

Educational Setting 

In addition to the common hardware implementation issues (e.g., technical specifications of the 
computers, connecting to a network,) there are a multitude of issues that must be addressed before students 
are allowed to use technology. The number and arrangement of computers in an educational setting is a 
critical issue facing the school districts. One arrangement is to have 25-30 computers networked in one 
room. This arrangement provides opportunities for equal access to the technology for all students. 
Depending on the software, teachers most likely are able to individualized the instruction to the needs of 
the students. In addition, if students are working in a computer lab, the physical setup will be of great 
importance as the layout of the room can directly influence the work of the students. For instance, if a 
classroom is set up in a U-shape students can view the status of their neighbor which can lead to many 
interesting research topics concerning competition. 

An alternative arrangement that is growing in popularity is to have 6-7 computers in the back of 
the regular classroom. Proponents state that this arrangement will provide teachers more opportunities to 
individualize the curriculum. On the other hand, some teachers in our project have expressed concern that 
the computers in the classroom arrangement will cause classroom management problems. They are 
concerned that teachers will not be able to provide equal access to the technology. If access is based on 
student-choice, students who are less computer literate or hold negative attitudes towards computers may 
benefit as much as computer enthusiasts. If access is teacher-driven, administering access time and 
curriculum covered may burden the teachers with additional administrative workload. Another issue is the 
amount of time the computers would be used. In our lab-based arrangement, computers are used almost 
constantly. Skeptics of the classroom-based arrangement, fear that the computers in the classroom will not 
be used as frequently by the students lowering the total access time to the available technology. 



Software 

Examining software further there are various questions about software variance teachers should be 
aware of when preparing to implement (i.e.. Can the software be installed to a network or does it only run 
on stand-alone machines?). Teachers should also ask whether or not the software has tools that set the 
curriculum and if these tools may be placed on additional machines besides the server on a network? 

Accompanying the questions of where the software is installed should be issues concerning the 
security of servers, teacher machines, and client machines. Teachers should address where the server will 
be kept and what access they will have to it during the day. Also, if teacher machines are available what 
restrictions and where will they be located should be noted. 

The topic of software and printers is another important issue. Teachers should know the software 
and be able to estimate the amount of paper which could be used and determine what is most important and 
should be included in the portfolio assessment. The use of printers, toner cartridges, and paper can become 
a costly issue if not addressed apriori. 

As more and more computers are wired for the internet new issues beyond the protection of 
computers for software viruses will become evident. Students having access to the internet must realize that 
they are accountable for the information they will be downloading. Many protective software packages are 
available to restrict student exploration, but even these have their limits. Issues might arise when the 
student is using the computer to do research and is searching for information on key words such as "breast” 
and "cancer". While many sites may have important and valued information concerning this topic, at this 
time search results may return sites that are inappropriate. Software which would prevent access to these 
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sites would also prevent access to health related sites as well because sites concerning questionable 
information would be blocked. 

Teachers will also have to ensure that students are accountable for information that is relevant and 
that students stay focused on the task at hand and not engage in viewing the latest information from a sports 
web site, playing a game, or sending e-mail. 



Technological Personnel Support 

An important factor in making implementation less intimidating is the school providing 
technological personnel support. These positions would serve to support teachers in the day to day 
maintenance of technology, communication, troubleshooting, and repair problems as they occur. Teachers 
may excel in understanding and using software to instruct students, but may not possess the knowledge or 
the experience to troubleshoot hard drive failure or install and run a file to update a version of the software. 

Schools may provide this support in the form of a Site Coordinator (SC). This position is usually 
filled by a teacher within the school and functions as a liaison between administration, teachers, students 
and vendors and helps with such duties as: facilitating testing, coordinating research activities, facilitating 
communication, scheduling site visits, conforming with district policy, and handling paper work (e.g., 
reporting hardware and software problems). 

Additionally, if funding allows, schools should hire a lab technician to manage the local area 
network, software, and computer hardware. This person should monitor classes coming into the lab and 
troubleshoot problems which may arise. A lab technician is usually a person with some computer 
experience, if not a computer programmer. This makes an ideal candidate to communicate with the vendor's 
technical support team. This type of experience can be very valuable and may help to avoid implementation 
issues before they originate. 

Lastly, the technological support personnel can investigate issues which may affect the 
implementation before the software is purchased. For instance, will the software work on both Windows 
3.1 and Windows 95? Where is the company headed? How does the company handle upgrades? When is 
the next upgrade of software due to be released? If software is purchased are upgrades free for a certain 
time? Is there a reduction in price? What is the price difference between a Site license and buying 35 copies 
for the lab? Has the company worked with education in the past? What issues did they face? 



Teacher, Administrator and Vendor/Research Project Responsibilities 

In order for technology to be successfully implemented and used effectively teachers, 
administrators, and the vendors/research projects must fulfill their responsibilities. Teachers ultimately 
control the curriculum and the use of the software as applied to the curriculum. They communicate with 
their Site Coordinator or the vendor concerning any problems which may occur with software, hardware, 
and students. Teachers also assist the SC in scheduling of calendar, dates, pretesting, site visits, entering 
student rosters, completing paper work (i.e., proprietary information), and possibly training new teachers. 
Teachers may also have to inform the administration of new technologies that exist and how significant it is 
that they are given an opportunity to teach using these tools and the importance that students are given a 
chance to learn using technology. It is also the responsibility of the teacher to discuss what goals the 
administration is trying to accomplish by installing new technology and how will they be asked to 
demonstrate the effectiveness of the software.. Teachers may also have to inform the administration what 
criteria is appropriate to judge the effectiveness of software besides cost effectiveness and increases in 
standardized scores. 

It is the responsibility of the administration to assist in finding and evaluating new software, and 
researching the software for potential use in the schools. The most important responsibility of the 
administration is to ensure teachers have time off for training, have time to learn the new software, and 
entrust technology to teachers that are accepting of technology. Lastly, it is the administrators obligation to 
know where to look for funding new technology, how to apply for these funds and to involve teachers by 
sharing this information with them. 

Perhaps the most overlooked responsibility in ensuring the success of technology is that of the 
vendor or the research project. Quiet simply it is their burden to demonstrate that their product effectively 
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teaches students. Vendors/Research projects should be able to present results and provide information 
about other groups who have used their products. Furthermore, vendors/research projects must provide 
users with materials and documentation guaranteeing that the information concerning underlying 
philosophies, project background, software and hardware requirements, communication, user rights, 
general help and instructions are available for reference. 

Research groups have additional responsibilities beyond vendors. For example, they must write 
the software so that no segment of their population is left unable to use it. Technical support to ensure 
continuity of the project must also be careftilly maintained. The research project must also report and 
publish results in a timely fashion and present results not only to their peers but must get this information 
out to the rest of the education community. 



Technical Support 

One of the most important aspects of any software purchase or involvement in a research project is 
the amount of technical support that is given or purchased. Teachers and administrators need to assess the 
potential amount and kind of technical support which will be needed (based on variables such as computer 
experience, technological support on site, existing hardware and software, etc. . . ) what type of technical 
support is preferred (phone support, e-mail support, web site support, site visits, etc. . . ) and the type of 
technical support available. Typically users are given a set of instructions, several floppy disks or a CD, 
and a phone number or web site in case of difficulties. Updates to the software or patches that fix problems 
in the software are often available as a download from the vendor's web page. Unfortunately, both 
scenarios may require an experienced computer user depending on the design of the computer lab (i.e. Will 
the software be installed to stand alone machines, or to a network?). 

Another important issue concerns the readiness of the site to receive technology support. 
Sometimes fixes to computer problems will require files to be sent to the vendor for investigation or files to 
be sent to the school via File Transfer Protocol (ftp). In other cases, some vendors may only provide 
technology support via e-mail or charge money for technical support provided over the phone. Typically, 
this is a charge per minute. In either case, the lab should be equipped with its own phone line for a modem 
or fax and a line that has long distance capabilities. 

To better provide the school with methods to resolve support issues it is very beneficial to have e- 
mail and ftp software. For instance, if a student cannot progress past a specific point in the software the lab 
technician can compress that student's directory and send it to the research group for troubleshooting and 
evaluation. Without electronic transfer of the files teachers must copy the files to a floppy and mail them. 



Evaluating the Effectiveness of Technology 

In the past, measuring success was a simple matter of counting the number of computers in a 
classroom and dividing that number by the number of students and reporting how the ratio of computers to 
students had advanced. In addition to this poor indicator, several evaluations of exemplary programs have 
had significant methodological problems and it is questioned if the results could be replicated on a large 
scale (Wenglinsky, 1998). In fact, a number of large-scale ITS evaluations have been viewed as failures 
because of flaws in the experimental design, and lack of planning and implementation problems (Shute & 
Regian, 1993). 

Recently the debate on technology's effectiveness has come into question. Criticism of technology 
has been flourishing in the media, and on Capital Hill, as lawmakers are considering cutting funds for 
technology (Trotter, 1998). Many people are asking if the investment of twenty years and billions of dollars 
has been worth it and the potential for backlash against school technology seems eminent. What is needed 
are several studies dealing with technological success in large field studies. 

When looking at software, teachers and administrators need to address the results the software has 
had in the past e.g.. What studies have been done? What were the research questions? What were the goals? 
Were the design and subjects listed? Were the results quantitative or qualitative? Is there anecdotal 
evidence given? 
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Summary 



Technology is not, itself, a panacea for the problems facing educators today. However, thoughtful 
implementation of instructional technology can provide schools and teachers with powerful tools for 
student learning and collaboration. 
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